Capturing text semantics for concept detection in news video

The overwhelming amounts of multimedia contents have triggered the need for automatic semantic concept detection. However, as there are large variations in the visual feature space, text from automatic speech recognition (ASR) has been extensively used and found to be effective to complement visual features in the concept detection task. Generally, there are two common text analysis methods. One is text classification and the other is text retrieval. Both methods have their own strengths and weaknesses. In addition, fusion of text and visual analysis is still an open problem. In this paper, we present a novel multi-resolution, multi-source and multi-modal (M3) transductive learning framework. We fuse text and visual features via a multi-resolution model. This is because different modal features only work well in different temporal resolutions, which exhibit different types of semantics. We perform a multi-resolution analysis at the shot, multimedia discourse and story levels to capture the semantics in news video. While visual features play a dominant role at the shot level, text plays an increasingly important role as we move from the multimedia discourse towards the story levels. Our multi-source inference transductive model provides a solution to combine text classification and retrieval method together. We test our M3 transductive model on semantic concept detection on the TRECVID 2004 dataset. Preliminary results demonstrate that our approach is effective.